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ABSTRACT 


The principal objective of this research is to demonstrate the capability of 
obtaining the acceleration in the navigation frame of a unistroke which can be written on 
any surface or in the air while correcting integration errors from the measurements of the 
IMU (Inertial Measurement Unit) of the pen-type input devices. With the core topic of 
obtaining the acceleration while correcting integration errors, there are four subsidiary 
research questions relating to the pen-type input devices. First question is how to segment 
a stroke from the tip of the pen-type input device movement of the user. Second question 
is how to adjust the integration errors rapidly growing as time increases. Third question is 
how to reconstruct 2-D trajectory from 3-D trajectory in the air. Fourth question is how to 


project the trajectory onto the x-y plane. 
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EXECUTIVE SUMMARY 


The objective of this research is to develop a 3-D pen input device that is able to 
track a person’s handwriting. With such a device, a person can write on a board, on a 
surface or in the air with no paper necessary and record the handwriting into a certain 
application which can be viewed through a graphical user interface. In order to 
accomplish this, a sensor will be attached to an existing writing instrument such as a pen, 
marker or a piece of stick. One application of such a device is for distance learning. 
When a teacher writes on the board in a classroom, his/her handwriting can be tracked in 
real time, and made available on the web to students at remote sites. Compared to other 
distance learning technologies such as video cameras, the 3-D pen input device would 


require much less bandwidth. 


The sensor used for developing the 3-D pen input device is called the nano 
Inertial Measurement Unit (nIMU) from MEMSense. The nano Inertial Measurement 
Unit is small enough to be implemented into a pen-like input device. The first step is to 
interface a nIMU to a computer so that the computer can read data from the sensor in real 
time. Second, the sensor provides acceleration, angular rate, and a magnetic field 
measurement in the sensor coordinates. Before integrating acceleration to obtain velocity 
and position, the acceleration measurements must be transformed into a fixed earth-based 
coordinate system. To do so, it is necessary to estimate the orientation or attitude of the 
sensor at any given moment. Stroke segmentation, zero velocity compensation, imaginary 
writing plane and projection, and rotation transformation will be investigated. Third, in 
order to integrate the acceleration in an earth coordinate system it is necessary to 
determine the start and end time of the integration, 1.e., determining the beginning and 
end of each writing stroke. Fourth, the computed hand trajectory from integration will 


contain errors, which should be removed as much as possible. 


In order to approach this research, several steps are taken to accomplish its 
mission. First, a nIMU will be attached to any writing instrument. After securing the 


sensor onto a writing instrument, data will be collected from the sensor. The data 


Xlll 


collected from the sensor are gyroscope, accelerometer, magnetometer, and counter for 
processing through three algorithms. The three algorithms are Quaternion, Factored 
Quaternion Algorithm (quaternion-based algorithm) and Tracking Algorithm 
Development. After processing and implementing the algorithms, the orientation of the 
pen will be estimated. The orientation represented in a quaternion will in turn be used to 


compute the position of the pen tip. 


The objectives of the research are accomplished by processing and implementing 
the algorithms through Matlab and C++ applications. In order to validate the final result 
of this research, a Matlab application was used for offline processing. In the Matlab 
application, the Factored Quaternion Algorithm was tested in static configurations to 
verify the correct results. After obtaining the correct result of quaternion in static 
configurations, a C++ application was used for real-time processing. The quaternion 
result was verified in dynamic configurations and agrees with that produced by the 


Matlab application. 
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I. FUNCTIONALITY OF 3-D PEN TYPE INPUT DEVICE 


A. PURPOSE OF PEN-TYPE INPUT DEVICE 


The purpose of using a pen-type input device is to improve communication with 
other people. Several types of people can benefit from the pen-type input device such as 
those in the defense community, distance-learning students, and/or hearing-impaired 
people. Here is an example of a scenario for each one of the three types of people that 


will use the pen-type input device. 


With the defense community, for instance armed forces out in the battlefield, if 
one soldier is not able to communicate with one another or their command leader or vice 
versa, without giving away their location to their armed adversary, the pen-type input 
device can help. With the pen-type input device, the soldier can write in the air and be 
able to get the word across to another soldier or command leader and save himself and/or 
another from their armed adversary. It is important for soldiers and command leaders to 


be able to communicate efficiently with each other. 


For distance-learning students who are not able to be present in the classroom, the 
pen-type input device can be quite handy for both the students and for the 
professors/teachers. In this case, the Naval Postgraduate School has distance-learning 
students from all across the country. With professors in the classroom, some of them will 
write vital information on the board to help the students understand the topic well. The 
distance-learning students would like to obtain the professor’s notes from the board. In 
order for this to happen, the professors might have to copy/scan their notes and upload 
them into Blackboard or any other form of eLearning portal. However with a pen-type 
input device, the professors are saved the trouble of either rewriting their notes or 
copying and scanning the notes then uploading them into a form of eLearning, then 
broadcasting them for the distance learning students. Instead, the professors can use the 
device in the classroom at the same time as writing on the broad for the distance-learning 


students and the students present in the classroom. 


Hearing-impaired students who are profoundly deaf or hard of hearing, with sign 
language as their primary language, attend a university/school with hearing people who 
might not have an efficient way to communicate with them in the classroom. With the 
pen-type input device, this will be an advantage to both parties in the classroom. The 
student is required to take notes in class but at the same time, miss some valuable 
information because he/she is not observing the interpreter while the professor/teacher is 
speaking and writing on the board. For the hearing students this is not a problem, since 
they can at the same time, listen and take notes without looking at the professor/teacher. 
With the pen-type input devices, the deaf/hard of hearing student can observe the 
interpreter while the professor writes on the board and the software will record every 
stroke of the professor handwriting on the board onto the screen of the computer and save 


it. 
B. COMPARSION OF PEN-TYPE INPUT DEVICES 


After searching to see what other pen-type input devices are available on the 
market, we found three pen-type input devices, EPOS Digital Pen & USB Flash Drive, 
Logitech Digital Pen and Fly Fusion. In order to justify the need for another device, we 
will compare and contrast between our pen-type input device and the other three 


currently on the market. 
1. EPOS Digital Pen & USB Flash Drive 


With this digital pen, a user can write, sketch and/or draw on any type of paper 
and it is portable. However, the user is required to attach the USB Flash Drive on the top 
of the paper which captures and stores any written text and/or drawings in real-time to the 
Flash memory, as shown in Figure 1. In addition to the digital pen, it requires its own 


refill ink which can be quite expensive. 





Figure 1. EPOS Digital Pen & USB Flash Memory From [1]. 


Zs Logitech Pen 


The Logitech pen, shown in Figure 2, works similar to the EPOS digital pen. It is 
portable and requires special ink. With the Logitech pen, the user is required to write on 
special paper, which is provided with the Logitech pen. However, it does not have a USB 


flash memory. 





Figure 2. Logitech Pen From [2]. 


3. Fly Fusion 


The Fly Fusion pen is a portable pen, which works similarly to the Logitech pen 
and requires writing on special paper. However, it does not have a USB flash memory to 
record the handwriting like the EPOS digital pen. The Fly Fusion pen has a tiny camera 
located at the tip of the pen, which reads the coordinates of the small dots that are located 


on the Fly Fusion paper. Figure 3 shows how the process works to accomplish it tasks. 


> 


Convert handwritten essays to digital 
text or write an email with pen and 
paper, Find important class notes 
using the key word search feature 


Customize your FLY Fusion Pentop 


Computer by using the FLY Vorld™ 
Application to download and manage 
the software that you need 


Use the FLY Fusion Notebook to 
capture your notes and get step-by-step 
help solving homework problems 





s 
s 


Figure 3. Fly Fusion Pen From [3]. 


4. 3-D Pen Input Device 


The proposed NPS 3-D pen input device used in this research contains some of 
the features described for the pen input devices above and important criteria which fit the 
user’s needs for utilizing the 3-D pen input device. The NPS 3-D pen input device 
contains a gyroscope to record the movement of the handwriting stroke, and the ability to 
attach itself to any writing equipment. Thus, this 3-D pen input device does not need 
special ink, paper, or camera to accomplish its goals. The 3-D pen input device, meets the 
needs of the defense community. In order to achieve the 3-D pen input device, a sensor 
which is obtained from a company called MEMSense [4] was attached to existing writing 


4 


instruments such as a pen, marker or a piece of stick. The MEMSense sensor used in this 
research is called nano Inertial Measurement Unit (nIMU). With this nIMU sensor which 
is small enough to attach to a writing instrument, a person will be able to record 
handwriting when it is being used in the air, on the board or on any surface. Using the 
nIMU sensor, the capability of obtaining the acceleration in the navigation frame of a 
unistroke, which can be written on any surface or in the air while correcting integration 
errors from the measurements of the IMU (Inertial Measurement Unit), of the pen-type 
input devices was demonstrated? With the core topic of obtaining the acceleration while 
correcting integration errors, there are four subsidiary research questions relating to the 
pen-type input devices. First question is how to segment a stroke from the tip of the pen- 
type input device movement of the user. Second question is how to adjust the integration 
errors rapidly growing as time increases. Third question is how to reconstruct 2-D 
trajectory from 3-D trajectory in the air. Fourth question is how to project the trajectory 


onto the x-y plane. 
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I. CRITERIA OF THE 3-D PEN INPUT DEVICE 


In order for the 3-D pen input device to accomplish its goals, there are six steps 
that need to be taken before utilizing the device. The first step is to interface the nIMU 
with a computer so that it will be enabled to read raw data from the sensor in real time. 
The second step is to obtain the accelerometer, gyro and magnetometer measurements in 
the sensor coordinate system and estimate the sensor orientation in quaternion form. The 
third step is to obtain the acceleration in earth coordinate system then integrates. The 
fourth step is to compute hand trajectory from the integration and remove any errors as 
much as possible. The fifth step is to display the final hand trajectory on the computer 
screen in real time as one writes on the surface or in the air with the pen input device. The 
sixth step is to make the tracking results available for others through the web. However, 
in this research we will focus mainly on the first three steps of the criteria. Step four 


through six will be performed in subsequent research. 


Within the first three steps, there are six sub steps to be performed before moving 
on to the fourth step. The first step is to obtain the accelerometer, gyro and magnetometer 
measurement in the sensor coordinate system, and to estimate the sensor orientation, is 
represented by a quaternion. The second step is to convert the accelerometer 
measurement from in the sensor coordinate system into the earth coordinate system using 
the quaternion rotation. The third step is integrating the accelerometer measurement 
which is in earth coordinate in order to obtain the velocity of the origin of the sensor 
coordinate system. The fourth step is to compute the velocity of the pen tip which is 
based on the origin of the sensor coordinate system and the angular velocity. The fifth 
step is to observe the pause phases in writing then apply the zero velocity correction. The 


sixth step is integrating the velocity of the pen tip in order to compute the 


position of the pen tip. Before proceeding with this, it is necessary to become well 
acquainted with the background of quaternion mathematics using the software called 
Matlab. The purpose of using the software Matlab is to be familiar with the quaternion 
and how to apply it in an algorithm. This is followed by implementing an algorithm using 
quaternion arithmetic which will track the movement of the unistroke. After becoming 
well acquainted with the algorithm which is in Matlab code, it will be converted into C++ 
code using the Microsoft Visual Studio 2005. 


A. INTERFACE nIMU WITH A COMPUTER 


In order to interface the nIMU sensor with a computer, there are two other pieces 
of equipment required for hardware setup and software installation; a power supply 
which is set to 8.3 Volts and a USB which connects the nIMU sensor to the power supply 
and to the computer. Figure 4 shows the hardware setup and in Figure 5 the software 


interfaces to obtain raw data from the nIMU sensor. 





Figure 4. Power Supply, nIMU and USB interface board connectivity From [4]. 


IMU Data Console (IDC) v7.1 
MAIN MENU 


Device: nIMU 
Protocol: I[2C 
Format: Units 
Print Counter: TRUE 
Gyro: 366 deg/s 

>: 5G 


. Auto-configure communications 
- Connect to device 

- Select device 

- Select protocol 

. Specify sensor ranges 

. Configuration Options 

- Refresh the screen 

. Exit 


a 
c 
d 
p 
¥ 
o 
f 
q 


Choice: 





Figure 5. nIMU Data Console Main Menu. 


In order to accomplish getting the right raw data from the right sensor, important 
details are needed. For instance, the sensor device which is used in this research is nIMU 
with the protocol I2C and set the data format in units along with the counter as shown in 


Figure 5. 


After the hardware setup and the installation of the software to work with the 
nIMU sensor, the next step is to establish the auto-configuration communication (a) of 
the nIMU sensor. The purpose of auto-configuration communication between the nIMU 
sensor and the computer is to ensure that the IDC.exe is able to identify the device. After 
it has been identified, then the device is connected to the computer as stated in the data 
console menu (c). This is the part where the user obtains raw data from the nIMU sensor. 


An example of the raw data is shown in Figure 6. 


aR VEPS) —6. 600640869 —6.68123596 —@. 600457764 - 48691223 
- 686183185 - 86196838 —8.00169863 —6. 600457764 46682068 
- 60311279 - 800640869 —6.600183185 —@.660411987 40741577 
- 600732422 - 00114441 6.06050354 —@. 600686646 - 68758732 
- 68218571 - 80382124 —6. 66058354 —8.060869751 —6. 60695861 
- 60160768 - 40141987 8 .80279236 -57764e-085 -8.00737 

- 60185286 - 608869751 6.060228882 - 800869751 -@.60714111 
- 80146484 - 808823975 —@. 600549316 - 80123596 - 88691223 
- 96178528 - 66850354 —-8.00268925 - 688549316 - 98741572 
- 60114441 —8.06201416 a SLSR PAGE Ri) - 000640869 - 68654682 
- 60151062 9.15527e-@85 —@ 60279236 - 900869751 - 6067749 

- 88362124 -57764e-885 8.800823975 - 90850354 - 08764465 
- 96215149 - 06151062 —8.008823975 - 66100788 - 66718689 
- 00128174 - 80160768 —6.60151062 - 80123596 - 60714111 
- 600549316 - 80178528 6.060646869 - 800778198 - 60758732 
- 88268925 - 000366211 —6 660549316 - 800778198 - 68718689 
- 88279236 - 808457764 @.8800326435 - 900686646 - 80695861 
- 60187683 - 80132751 —-6. 60196838 - 008915527 - 80672913 
- 80362124 - 80247192 —-6.60261416 - 000686646 - 88759888 
- 98192261 - 868915527 —8 008595093 - 660640869 - 88723262 
- 60183185 - 00151862 6.060549316 - 800778198 - 88723267 
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Figure 6. Raw Data obtained from nIMU sensor. 


Based on the IDC (IMU Data Console) manual of the nIMU sensor device, there 
are a total of 13 columns of raw data. The first column is the counter (time). The second 
to fourth columns are the gyroscope xyz coordinates, fifth to seventh columns are the 
acceleration xyz coordinates, and the eighth to tenth columns are the magnetometer xyz 
coordinates. The last three columns are the temperature of the xyz gyroscopes. This 
experiment does not use the gyroscope temperature; however, it will be available for 
future researchers. These raw data can be saved as a text file through the main menu 
option (0) configuration. The purpose of saving the raw data into text file was to use the 
algorithm with quaternion arithmetic and learn how to interact with it in offline 


processing. 
B. PROCESS RAW DATA THROUGH MATLAB 


After testing the nIMU sensor by moving the device by the movements of yaw, 
roll and pitch, all the raw data from the sensor was stored into a text file. The core of the 
experiment is being able to work with the raw data from the sensor and process the data 
through Matlab using an algorithm. In order for the user to use the raw data obtained 
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from the sensor of the nIMU, the user needs to implement an algorithm which is called 
the Factored Quaternion Algorithm that will interact with the data from the sensor. The 
algorithm, which will work with the raw data, will be based on a concept called 


quaternion. 
1. Quaternion 


Before going any further into the factored quaternion algorithm, an understanding 
of the quaternion is important. The quaternion is “a non-commutative extension of 
complex numbers” which was first invented by an Irish mathematician Sir William 


Rowan Hamilton [5]. In quaternion algebra we define three elements i, j, k with the 





properties i? = j? =k? =ijk =—1 [7]. Based on this, as an extension of complex 
members, a quaternion is defined as a+bi+cj+dk, with a, b, c, and d real numbers. 
Within the quaternion arithmetic, we define an addition, multiplication, product, norm, 
complex conjugate, inverse and the rotation matrix, which we use to express the location 
and motion of the sensor. Below are the general formulas to work with quaternion 


arithmetic: 


a. Quaternion Addition 


P= Py +P, + jp, + kp ; ; 

7 : : ; : : P+ =(PotQ) tii +H) + IP2 +) + kK(P3 +4) 
= +14, + Jd, + kas 
b. Quaternion Multiplication with Scalar 


d=) + ig, + jd, +kq 
athena i cq = (cq) + i(cg,) + (eqn) + k(cqs) 


c = scalar 
Cc. Vector Cross Product 
EO ick 
PXQ=|P, Po P3)=1(P293 — P42) + I(Psh — Pids) + (Dido — Po) 
4h 
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d. Vector Dot Product 
P°d = Pid, + Prdr + P39 
e. Quaternion Multiplication with Two Quaternions 


When multiplying two quaternions together, the dot product and cross 


product are used to obtain the product of two quaternions. 


Note: p=ip,+ jp,+kp, and q=ig,+ jq,+kq, 


P=P.+P Bh 2 fiy ty & Se 
~ (PI = Podo — P°?A+ Pod + HP + PX 
q=+4 
f. Quaternion Norm 
lj=V40 +4 +4 +45, 
g. Quaternion Complex Conjugate 


q =4) +19, + iq, + ka, 
q = dy — 19, — JQ, — kay 


h. Quaternion Inverse 
Ses de 
qs Tae 
| 
i. Rotation Matrix From Quaternion 


2qn(-N+2q; 24,4,-249, 24,4, +2404 
R= 24,4 + 2.4593 2q;(-1) +243 24593 — 24% 
24193-2409, 29,4;+2409, 245 (-1) +295 
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2. Factored Quaternion Algorithm 


After understanding the background in quaternion arithmetic, we introduce the 
factored quaternion algorithm. Matlab software was used to implement the quaternion 
formulas in offline processing before moving on to C++ in real-time processing. In order 
to implement an algorithm which uses quaternion formulas, there are three positions that 
use the quaternion formulas; elevation quaternion, roll quaternion, and azimuth 
quaternion along with the singular avoidance. These positions apply the same concept as 
flight dynamics [6] using the orientation and control in three dimensions. Instead of using 
the flight itself, the usage of the nIMU sensor will be in replacement for the flight. There 
are three critical nIMU sensor dynamics parameters, which are the angles of the rotation 
in three dimensions; pitch, roll, and yaw. To help understand the whole picture of the 
usage of pitch, roll, and yaw in nIMU sensor, Figure 7 shows the coordinates of a nIMU 


sensor. The z-axis is the yaw, the y-axis is the pitch and the x-axis is the roll. 
+X-axis Acceleration 


+X-axis Angular Rate (CCW) 
+X-axis Magnetic Field 





Figure 7. Pitch, Roll, Yaw with nIMU sensor From [4]. 


a. Elevation Quaternion 


The elevation quaternion is a pitch rotation of a rigid body about the y- 


axis by an angle@, which is aligned with the Earth coordinate system. Since the x-axis 


13 


accelerometer is perpendicular to gravity, it has zero acceleration at rest. Whereas the y- 
axis accelerometer also has zero acceleration, the z-axis accelerometer shows —g 


(gravity). Including the elevation, the computation will be the following: 


X-axis accelerometer: 
a, = gsin@ 
z-axis accelerometer: 


a,=—gsin@ 


ad m 
g=-9.81 ie 


where is the acceleration due to gravity. 


The measured acceleration vector in the body coordinate system is 


a 
denoted as z 


The acceleration vector can be obtained from the raw data in column 
fourth to sixth of Figure 6. After obtaining the acceleration vector, it needs to be 
normalized to a unit vector. Therefore, the normalized vector of the acceleration 


measurements is: 


ey 


g] 
II 
| 

Q 


whereas la| is the norm of the acceleration vector a. Due to 


the convention restricted from the elevation angle @, the range must be in the interval 
By <O0< yy, and cos@ must be positive. To obtain the value of cos@ and sin@, the 


formulas below show the computation which are needed for the next computation. 
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sin? =a, 
cos@ = V1—sin’ 6 


In order to obtain the elevation quaternion, there are two other values that 
are needed before processing the elevation quaternion. The two values, which are needed, 


are the half-angle formulas as shown below: 


sins = sign(sin 9),/(1—cos @) / 2 


where sign() return the +1 for positive and -1 for negative 


cos = (I +e086)72 


After all the computation above, the final elevation quaternion formula is: 


qd = cos 5 000)+ sin (0 010) 


b. Roll Quaternion 


The roll quaternion is the roll rotation of a rigid body about the x-axis by 
an angle @. The acceleration that is measured by the z-axis accelerometer with roll angle 
@ is assigned to zero given by the equations of a, =—gsin@ [6]. The formula for the y- 
axis accelerometer is given due to the fact that the azimuth does not change the 


measurement. Therefore the equation for the y-axis accelerometer is a, =—g cos@sing 


and the z-axis accelerometer is a, =—gcos@cos@. After obtaining the results of the y 


and z-axis accelerometer, it needs to be normalized. Therefore the normalized 


acceleration measurements are: 


a, =—cos@sing 


a, =—cosOcos¢@ 


The value of cos@ comes from 
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sind =a, 

cos = V/1—sin’ 0 

However, if the value of cos@ is not zero, the value of sing and cos gare 
computed from 


sin g =—@, /cos0 


cos ¢=—a,/cosO 


If the value of cos@is zero then the x-axis of the body coordinates is 


vertically oriented [6]. Due to the convention restricted from the roll angle¢, the range 
must be in the interval -7 <@< 7. In obtaining the half angle values for@, this can be 


computed in the same manner as 0: 


sin’ = sign(sin g),/(1—cos g)/2 


where sign() return the +1 for positive and -1 for negative 


cos = (+ e059)72 


After obtaining all the values ford, the roll quaternion equation is 
computed by: 
fe @ ae 
q, =cos—(1000)+sin—(0 1 0 0) 
2 2 
c Azimuth Quaternion 


With azimuth rotation, it is the yaw rotation of the rigid body about the z- 
axis by an angle '¥ . Therefore, azimuth rotation has no effect on the roll and elevation 
quaternions; so in order to estimate the azimuth quaternion, the roll and elevation 
quaternions need to be computed first. They are then used to rotate the normalized 
magnetic field measurement vector. Before normalizing the magnetic field measurement, 


it is necessary to obtain the magnetic field measurement from the raw data in column 
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seventh to ninth. The normalized magnetic field is computed in the same way as the 


mM, 
. . b m b 
normalized acceleration field: °m = in =| “m, 
m 
b 


With this measurement, the normalized magnetic field in body coordinates 


is computed in the Earth coordinate system as follows 
'm = q. q, bm Ca 


The final azimuth quaternion is then computed as 
, =cos (I 00 0) +sin=(0 001) 


The magnetic field measurement vector in earth coordinate should agree 


with the known local normalized magnetic field vector with ” the azimuth angle: 


n,| |cosy —siny “mM, 
n, i. siny cosy || “m, 


field N and the normalized magnetometer field measurement M are defined as 


The | normalized local magnetic 


From the definition of the magnetic field measurement vector in earth 


coordinate and the local normalized magnetic field vector with “ the azimuth angle, we 


can relate N and M as 
N,| |cosy —siny || M, 
N,| |siny cosy || M, 
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From this we obtain °°S¥ and siny as 
cosy | _ M. M,|N, 
siny | |-M, M,||N, 


cos a sin a 


In order to obtain 2 and 2 before calculating the azimuth 


quaternion, we use the same equations as previously: 


sin = sign(siny),/(1—cosy) /2 


where sign() return the +1 for positive and -1 for negative 


cos = y+ cosy)/2 


From these equations, we compute the orientation of the rigid body as 


q =, q. q, . 
d. Singularity Avoidance 


With the factored quaternion algorithm, we use three angles to derive the 
quaternion estimate of the orientation of the rigid body. However, with any three- 
parameter representation of a 3-D orientation, it is known that it is impossible to avoid 
singularity. Therefore the factored quaternion algorithm has a singularity and we need to 
implement a technique to avoid it. In order to do this we need to understand where the 
singularity occurs in the factored quaternion algorithm. The singularity occurs when the 
elevation angle is + 90° and this happens when the cos 9 = 0 or a, =0 in 

sin ¢ = —a, /cos 0 


cos¢ = -a,/cos@ 


To avoid the singularity, we needed to check the value of a,. If the value 


of a.<eé, withe a predefined constant such as ¢=0.1, then the calculation of the 
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normalized acceleration measurement and the normalized magnetic field measurement 


vector are needed. The following is the calculation to be done: 


— eye 
Corser =n A Ty 


b _ kb -1 
MN ofset =a My 


where q, is the offset (rotation) quaternion defined as 


Qe =cos$(1 00 0) +sin (0 010). 
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Ul. TRACKING ALGORITHM DEVELOPMENT 


All of the important content from quaternion arithmetic and factored quaternion 
algorithm comes together into the tracking algorithm development. This is the final stage 
of obtaining the outcome of what is expected of the input device in offline processing 
through Matlab. However, it is not the final stage of the whole research plan because the 
main purpose of this research is able to obtain the outcome in real-time processing and 
this will be accomplished using C++. Within the tracking algorithm development, there 
are six steps which need to be accomplished in obtaining the outcome. After these steps, 
which will be explained shortly, have been accomplished, the outcome of the input 
device should be a graph of what the user has written with the sensor. Figure 8 shows 
where the sensor is supposed to be located on a pen or marker. As you can see there are 


three separate coordinates, one is for the sensor with x, y, z,, second is for the pen or 


so? 


and the third is the Earth-fixed coordinates withx, y, z,. The 


e 


marker tip with x, y, z,, 


purpose of having the Earth-fixed coordinates is to estimate the position of the pen or 


marker. 





Figure 8. Earth-Fixed Coordinates x, y, z,, Sensor Coordinates x, y, z,, and Pen or 


marker Tip Coordinates x, y, z, From [10]. 
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A. SENSOR ORIENTATION 


After obtaining the accelerometer and magnetometer measurements, we process 
them through the factored quaternion algorithm (FQA). However, the FQA does not 
include the gyro measurements which are obtained through the raw data of the nIMU 
sensor itself. All three measurements are in the sensor coordinate system and the next 
step is to estimate the sensor orientation. To estimate the sensor orientation, the 
orientation will be represented by a quaterniong . The sensor orientation will be the same 
as the writing instrument orientation because the sensor itself will be rigidly attached to 


the pen or marker as shown in Figure 9. 





Figure 9. MEMSense nIMU attach to a pen or marker From [10]. 


B. EARTH COORDINATE SYSTEM 


With the accelerometer measurement in the sensor coordinate system which was 
obtained through the FQA, the measurements are required to be converted into earth 


coordinate system to determine the location of the pen or marker and the sensor attached. 


This can be accomplished by the quaternion rotation operator, ‘a=q “a q where ‘a is 
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the accelerometer measurement in earth coordinate system, “a is the accelerometer 
measurement in sensor coordinate system, gand q° are the quaternion and quaternion 


conjugate which are obtained from the FQA. 
C; VELOCITY OF SENSOR 


In order to obtain the velocity of the sensor shown as point A in Figure 8, it is 
needed to integrate the accelerometer measurement in earth coordinate. This yields the 


velocity of the sensor at point A as the three-dimensional vector in earth coordinate 


system. The velocity of point A in earth coordinate is given by: ‘v, = | (“a)dt whereas 


“a is obtained by the quaternion rotation operator. 
D. VELOCITY OF PEN BASED ON SENSOR 


After computing the velocity of the sensor in point A, the next step is to compute 
the velocity of the pen or marker based on the sensor in point A and the angular velocity. 
In order to accomplish this calculation, there are some components which need to be 
computed before reaching the final calculation of the equation 


Uy = Vere COS Pie) a which is the velocity of the pen or marker based on the 


sensor. Note the “s” which stands for “sensor” and “e” which stands for “earth”. The first 


step is to obtain the @ which is the angular rate measurement from the gyroscope in 


column two through four from the raw data of the sensor. The Pas is the distance 


between the nIMU and the pen or marker tip as shown in Figure 8. In order to obtain the 


Pas , there are two calculations which are needed. The first calculation is the So which is 


the skew symmetric matrix associated with the @, the angular rate measurement. The 


: ee 2 
skew symmetric matrix “@ is given by: 


0 -a, 2, 
S,=| @ 0 -a, 
-O, @, 0 


The skew symmetric matrix will be stored in the S array to go through enumerate 
of the raw data of the sensor for the gyroscope. The S array will be in a3”*3 matrix by 


5, (1) 
S=| 4 
5,(”) 


stacking n skew symmetric matrices given by: . The second calculation is the 


YA which is the velocity of point A in the sensor coordinate, as shown in Figure 8 and it 


is given by 
‘Vv. @O=¢ WV, Mad 
where i = the number of row. 


The term "4 is obtained by integration as discussed earlier. The purpose of doing 
the calculation for "A through "4 @) is to convert from earth coordinate back to sensor 


coordinate. The Va will be a 3”! vector by stacking the velocities of point A given by: 


v0) 
Va = = 


‘v4 (1) 
After obtaining the calculation of both the ‘(skew symmetric matrix) and 
wa (velocities of point A), we obtain the distance between the nIMU and the pen or 


_ el o-loTs 
Pap =(SS) SS" V, where T is the 


marker tip, Pas . The equation is given by: 
transpose of the matrix S. With the final calculation for Pas | we compute the vector 


cross product with ‘ and the result is in sensor coordinate system. To convert the result 


from sensor coordinate system to earth coordinate system, quaternion rotation operator is 


applied along with the final calculation of Ve Mahe CO Pin) a : 
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E. DETECT PAUSE PHASES 


When the user is writing something, the sensor is recording every unistroke of the 
data. However, the user is not always writing and there is a pause in between two 
unistrokes. The next step is to be able to detect this pause. If there is a pause phase, it is 
necessary to apply zero velocity correction to make sure that there is no data running 


during the pause phase. 
F. POSITION OF THE PEN TIP 


The final step of the tracking algorithm development is to compute the position of 
the pen tip. In order to compute the position of the pen tip, it is necessary to integrate the 


velocity of the pen tip, 
“Va = “Va tq COX "Paz) q 


The equation for computing the position of the pen tip is given, 


“Py =| (‘vp)dt 

However, the pen tip position, Po , 1s computed in three dimensions. Because the 
user normally writes on surface such as a piece of paper on a table, or on a blackboard, 
the pen tip trajectory is actually in two dimensional plane but not always. For instance, 
the user might write in the air and there is no plane to be able to hold any hand motions 
in. With the calculation of the position of the pen tip, which is the result in three 
dimensional, it will give the final result of the handwriting tracking. However, the results 
will be projected onto a two-dimensional space in order to be displayed on a computer 


screen. 
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IV. RESULT OF RESEARCH 


After coding all the algorithms for quaternion, factored quaternion algorithm, and 
tracking algorithm development in Matlab for offline processing, we tested the sensor’s 


raw data to see the result of what is expected. In order to accomplish the result of what is 


expected, we ran the factored quaternion algorithm first. We set the sensor pointing to 


North direction and sitting on a flat surface as it is shown in Figure 10. 





Figure 10. MEMSense nIMU sensor and Compass set in North direction. 


The purpose of setting the sensor in North direction is due to the fact that we want 
the accelerometer xyz coordinates to be ideally 0 0 1 and magnetometer xyz coordinates 
to be ideally zero in y component and nonzero in x and z components. This is true if the 
sensor is motionless and in reference to the gravity of the earth pointing downward. 


While the sensor is in North direction, we obtain the raw data as shown in Figure 11. 


2] 


Figure 12 is the same raw data obtain from the nIMU sensor but in a readable format. The 


bottom of Figure 12 is a row of data showing accelerometer and magnetometer 


measurements. 


Figure 11. 
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Raw Data with Sensor point in North direction. 
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XYZ 





Figure 12. Readability format of the Raw Data. 


When the sensor’s accelerometer and magnetometer xyz coordinates are what is 
expected, we implemented this data into the factored quaternion algorithm. Ideally, the 
result of the quaternion should be 1+0i+0j+0k. After running the factored quaternion 


algorithm, the result of the quaternion is shown in Figure 13. 


quaternion = 


O.99707658623564 —-O.00475008895120 0.01001560500450 O.0756003 0086997 


Figure 13. Matlab Application of Quaternion Result. 


With real time processing which is performed through C++ application, the 
algorithms are the same except for the coding style and how the result will be presented. 
With offline processing as mentioned before, the result will show after the calculation of 


the factored quaternion algorithm as shown in Figure 13. With real time processing, 
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while the sensor is running, at the same time the factored quaternion algorithm through 
the C++ application will obtain the raw data, calculate the quaternion and display the 
result on the application. If the sensor is moving in different direction, a different 
quaternion result will be displaying on the application. Figure 14 shows the result of the 
C++ application when the sensor is pointed to North direction. The result agrees with 


that produced by the Matlab program. 


”« MEMS_IDC - Microsoft Visual Studio 
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Figure 14. C++ Application of Quaternion result. 
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V. CONCLUSION & RECOMMENDATIONS 


A. CONCLUSION 


The objective of this research was to read the unistroke of handwriting by 
developing a 3-D pen input device. The emphasis was on implementing algorithms for 
tracking orientation of the 3-D device. First, a Matlab application was developed for 
offline processing. However, this is not the end result of the research plan for the 
development of the 3-D device. The main purpose for this research plan is to perform this 
experiment with real-time processing. In order to read the unistroke in real-time 
processing, a C++ version of the factored quaternion algorithm was implemented. With 
the C++ version of the algorithm of the 3-D input device, the user can write on any 
surface or in the air and have his/her handwriting recorded. As soon as the research is 
finalized, several types of people will be able to benefit from the development of the 3-D 
input device. Types of people include, but are not limited to, the defense community, 


distance-learning students, professors/teachers and hearing-impaired individuals. 
B. RECOMMENDATIONS 


Although orientation tracking of the 3-D input device was implemented and tested 
in this research, position tracking has not been completed. For future investigation 
relating to this research, it is recommended that a position tracking algorithm be 
developed and implemented. After the completion of the position tracking algorithm, it is 
possible to collect the data and exhibit the result through a graphic user interface (GUI) to 
display handwriting. 
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